X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7232E.A6AE27C8@onstor-exch02.onstor.net>; Mon, 18 Dec 2006 21:29:27 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7232E.A6AE27C8"
Content-class: urn:content-classes:message
Subject: RE: Clio: What's the update on the stop-ship defects?
Date: Mon, 18 Dec 2006 21:29:26 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E01CC2C3E@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E0116B50A@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Clio: What's the update on the stop-ship defects?
thread-index: AccjHkhEfuffC80oQ/WGN6xikiLpSQAD7nXw
From: "Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>
To: "Sandrine Boulanger" <sandrine.boulanger@onstor.com>,
	"dl-Clio" <dl-Clio@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7232E.A6AE27C8
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Regarding #16898:

If the problem happens again, collect outputs of "fscmd errstats" and
"stat show ldavg".
Save the elogs at that time.  Collect the stack traces of all threads
running on FP by connecting to gdb.
Looking at some EVM/IO stats also would be useful, but I am not sure
what all commands we have for that.

Regards,
Jobi

_____________________________________________
From: Sandrine Boulanger=20
Sent: Monday, December 18, 2006 7:32 PM
To: dl-Clio
Subject: Clio: What's the update on the stop-ship defects?

#16898, ops to 0 on cheetah soak
Cheetah soak is still running and I'm monitoring the stats every 5
minutes, so that we can figure out at what time the ops drop if they
ever do again.

| -N/A-          |   7665 |  15454 |   7665 |  15453 |    2.98M |
5.66M |   31.10M |    5.61M |
| -N/A-          |   9516 |  15108 |   9516 |  15108 |    4.56M |
6.13M |   32.72M |    5.41M |
| -N/A-          |   9675 |  14648 |   9675 |  14649 |    3.77M |
5.41M |   31.91M |    5.44M |
| -N/A-          |  10350 |  14549 |  10350 |  14549 |    4.00M |
5.50M |   34.48M |    5.64M |

I'm not sure if I should crash TXRX or FP when this happens, or which
other stats should be collected, any suggestion is welcome.
The defect has been updated but there's no clear lead. No update on the
analysis of vsvr requests.

#?, snmpd and fp/txrx crashes on MD
What's the latest status on the system upgrade investigation?

------_=_NextPart_001_01C7232E.A6AE27C8
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7650.28">
<TITLE>RE: Clio: What's the update on the stop-ship defects?</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Regarding =
#16898:</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">If the problem happens again, collect outputs =
of</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">&#8220;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">fscmd errstats</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">&#8221;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial"> and</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">&#8220;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">stat show ldavg</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">&#8221;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Save the elogs at that time.&nbsp; Collect the stack =
traces of all threads running on FP by connecting to =
gdb.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Looking at some EVM/IO stats also would be useful, but I =
am not sure what all commands we have for that.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Regards,</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jobi</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Monday, December 18, 2006 =
7:32 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Clio<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Clio: What's the update on the stop-ship =
defects?</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 FACE=3D"Arial">#16898, ops to 0 on =
cheetah soak</FONT></B></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 FACE=3D"Arial">Cheetah soak is still running and I&#8217;m =
monitoring the stats every 5 minutes, so that we can figure out at what =
time the ops drop if they ever do again.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">| =
-N/A-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
|&nbsp;&nbsp; 7665 |&nbsp; 15454 |&nbsp;&nbsp; 7665 |&nbsp; 15453 =
|&nbsp;&nbsp;&nbsp; 2.98M |&nbsp;&nbsp;&nbsp; 5.66M |&nbsp;&nbsp; 31.10M =
|&nbsp;&nbsp;&nbsp; 5.61M |</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">| =
-N/A-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
|&nbsp;&nbsp; 9516 |&nbsp; 15108 |&nbsp;&nbsp; 9516 |&nbsp; 15108 =
|&nbsp;&nbsp;&nbsp; 4.56M |&nbsp;&nbsp;&nbsp; 6.13M |&nbsp;&nbsp; 32.72M =
|&nbsp;&nbsp;&nbsp; 5.41M |</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">| =
-N/A-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
|&nbsp;&nbsp; 9675 |&nbsp; 14648 |&nbsp;&nbsp; 9675 |&nbsp; 14649 =
|&nbsp;&nbsp;&nbsp; 3.77M |&nbsp;&nbsp;&nbsp; 5.41M |&nbsp;&nbsp; 31.91M =
|&nbsp;&nbsp;&nbsp; 5.44M |</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">| =
-N/A-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |&nbsp; =
10350 |&nbsp; 14549 |&nbsp; 10350 |&nbsp; 14549 |&nbsp;&nbsp;&nbsp; =
4.00M |&nbsp;&nbsp;&nbsp; 5.50M |&nbsp;&nbsp; 34.48M |&nbsp;&nbsp;&nbsp; =
5.64M |</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">I&#8217;m not sure if I should crash TXRX or FP when this =
happens, or which other stats should be collected, any suggestion is =
welcome.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">The =
defect has been updated but there&#8217;s no clear lead. No update on =
the analysis of vsvr requests.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 FACE=3D"Arial">#?, snmpd and fp/txrx =
crashes on MD</FONT></B></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 FACE=3D"Arial">What&#8217;s the latest status on the system =
upgrade investigation?</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

</BODY>
</HTML>
------_=_NextPart_001_01C7232E.A6AE27C8--
